Add an evaluation suite for the Workshop agent - #271

Adds real end-to-end evaluations for the Workshop agent. Stacked on #270, which supplies the shared workerd and Cap'n Web driver.

Execution

Each trial gives the production agent a natural-language task in a real Workshop. The agent builds a Gadget, and the verifier calls that Gadget's real RPC. The same scenario definitions run against:

• local workerd, for iteration and repeated experiments
• an Access-authenticated deployed preview, for functional end-to-end validation

Run locally with pnpm evals. Set WORKSHOP_EVAL_TARGET to a preview router URL and provide CF_ACCESS_TOKEN to target a deployment.

Live cases use evals/**/*.eval.ts. The package's cached test task includes only src/**/*.test.ts, so pnpm test never invokes a model. The root eval command invokes the live package directly after the build, preserving credentials outside Vite+ task caching.

Framework ownership

The package uses vitest-evals for normalized transcripts, tool calls, errors, usage, timings, scores, the local report UI, and combined GitHub reporting. Repository code is limited to the Workshop target bridge, Gadget RPC verifier, and task content. There is no custom statistics package, report format, trace UI, or shard reducer.

Initial scenarios

• appointment capacity under concurrent booking
• a multi-turn household ledger with exact arithmetic and preserved earlier behavior
• creation and editing of a real standard Doc

Checks score observable behavior, not a prescribed implementation technique. Tool failures, agent errors, turns, token usage, cost, and wall time remain diagnostics.

Automation

The Workshop evals workflow has only workflow_dispatch. It never runs on pull requests, pushes, merges, or a schedule. It uses existing AI Gateway secrets, native Vitest sharding, and the pinned getsentry/vitest-evals action.

Live proof on current main

• Appointment desk: GLM 5.2 passed every check; score 1.00, 68,194 tokens, 27 tools, 805 seconds.
• Expense ledger: the current git-storage path completed both turns and the turn-two check proved turn-one state survived. The agent then failed to recover from one malformed editFile call, so the new budget methods were absent: 7/9 checks, score 0.78, 29,354 tokens, 13 tools. This is an agent failure recorded by the suite, not hidden as a harness pass.

Verification

• pnpm build
• pnpm test
• pnpm lint
• 19 eval-package unit tests pass